feat(accuracy): AIME 2025 lighteval-backed benchmark (AIP-876)#926
Merged
FrankD412 merged 1 commit intoMay 29, 2026
Merged
Conversation
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@3e85d67ae69a5294c4db72891ab72f09f9f6a2feRecommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@3e85d67ae69a5294c4db72891ab72f09f9f6a2feLast updated for commit: |
This was referenced May 12, 2026
Contributor
Author
Stack dependencyThis PR is part of an 8-PR stack aligning aiperf's accuracy benchmarks with Merge order:
This PR: position 5 of 8 — base branch is After each upstream PR merges, the downstream PR's branch will be rebased |
debermudez
force-pushed
the
dbermudez/aip-875-implement-aime24-benchmark-loader
branch
from
e0576be to
cd239a5
Compare
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
358d5bd to
9bbe752
Compare
debermudez
force-pushed
the
dbermudez/aip-875-implement-aime24-benchmark-loader
branch
from
cd239a5 to
ed0edf6
Compare
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
9bbe752 to
599d8f4
Compare
debermudez
force-pushed
the
dbermudez/aip-875-implement-aime24-benchmark-loader
branch
from
ed0edf6 to
9eabc25
Compare
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
599d8f4 to
d7552b6
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
d7552b6 to
b8645f1
Compare
debermudez
force-pushed
the
dbermudez/aip-875-implement-aime24-benchmark-loader
branch
from
9eabc25 to
05ece47
Compare
debermudez
force-pushed
the
dbermudez/aip-875-implement-aime24-benchmark-loader
branch
from
05ece47 to
80e91dc
Compare
Base automatically changed from
dbermudez/aip-875-implement-aime24-benchmark-loader
to
main
May 29, 2026 00:25
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
b8645f1 to
62d9fbe
Compare
Implement ``AIME25Benchmark`` mirroring the trt-llm benchmark recipe's ``acc_bench_lighteval.py:aime25`` configuration: same ``aime_prompt_fn`` zero-shot rendering, ``generation_size=32768``, ``hf_repo="yentinglin/aime_2025"``. Same shape as ``AIME24Benchmark`` just pointed at the 2025 mirror. The loader emits one ``BenchmarkProblem`` per dataset row with the bare problem text as ``prompt``, ``str(answer)`` as ``ground_truth``, and ``metadata.generation_size`` = 32768. ``tasks`` / ``n_shots`` / ``enable_cot`` are accepted for protocol uniformity but ignored. Pair with ``LightevalExprGrader`` for the recipe's ``expr_gold_metric`` extraction. Built on top of AIP-875 (lighteval sub-stack ordering: 875 → 876). No heavy optional dependency — ``datasets`` is core — so CI gets 100% line + branch coverage out of the box. Updates the stub registry: drop ``aime25`` from ``test_accuracy_config.STUB_BENCHMARKS``, drop ``is_implemented: false`` from the ``aime25`` plugins.yaml entry, switch ``default_grader`` to ``lighteval_expr``, add the ``aime25`` row to ``docs/accuracy/accuracy-benchmarking.md``, and move it from "Still Stubbed" to "Implemented" in ``accuracy_stubs.md`` (refreshing the Status Summary, Method Count Summary, and Suggested Implementation Order accordingly). Signed-off-by: Elias Bermudez <dbermudez@nvidia.com>
debermudez
force-pushed
the
dbermudez/aip-876-implement-aime25-benchmark-loader
branch
from
62d9fbe to
3e85d67
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
💤 Files with no reviewable changes (1)
✅ Files skipped from review due to trivial changes (2)
🚧 Files skipped from review as they are similar to previous changes (3)
WalkthroughImplements the AIME25 benchmark loader (lighteval-aligned) to load ChangesAIME25 Benchmark Implementation
🎯 3 (Moderate) | ⏱️ ~25 minutes
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/aiperf/accuracy/benchmarks/aime25.py (1)
49-49: ⚡ Quick winRemove type annotation from
**kwargs.The
**kwargs: Anyannotation should be removed. Based on learnings, variadic keyword arguments should remain untyped unless explicit named parameters are needed.♻️ Proposed fix
- def __init__(self, run: BenchmarkRun, **kwargs: Any) -> None: + def __init__(self, run: BenchmarkRun, **kwargs) -> None:Based on learnings: "In Python projects (e.g., in aiperf), avoid adding type annotations to **kwargs like **kwargs: Any. The variadic keyword arguments are inherently dynamic; leave **kwargs untyped or replace with explicit, named keyword parameters if a concrete contract is needed."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/aiperf/accuracy/benchmarks/aime25.py` at line 49, The __init__ signature for the class in aime25.py annotates variadic keywords as **kwargs: Any; remove the type annotation and change the signature to use untyped **kwargs so it reads def __init__(self, run: BenchmarkRun, **kwargs) -> None:, updating any references to __init__ if they assert the typed form and ensuring no additional named keyword parameters are required; this removes the unnecessary **kwargs: Any annotation while preserving behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/accuracy/accuracy_stubs.md`:
- Line 328: The docs currently give conflicting guidance for math_500 (which
mirrors AIME24Benchmark) by recommending pairing with lighteval_latex while
earlier listing math_500's default grader as math; update the accuracy_stubs
entry for math_500 to explicitly state that pairing with lighteval_latex is a
planned post-implementation transition (or else change the recommended grader to
match current default 'math') so contributors have a single source of truth;
reference the symbol math_500 and the grader names lighteval_latex and math when
making this clarification.
---
Nitpick comments:
In `@src/aiperf/accuracy/benchmarks/aime25.py`:
- Line 49: The __init__ signature for the class in aime25.py annotates variadic
keywords as **kwargs: Any; remove the type annotation and change the signature
to use untyped **kwargs so it reads def __init__(self, run: BenchmarkRun,
**kwargs) -> None:, updating any references to __init__ if they assert the typed
form and ensuring no additional named keyword parameters are required; this
removes the unnecessary **kwargs: Any annotation while preserving behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0676f619-de8e-4ffa-acb4-481290ad2309
📒 Files selected for processing (6)
docs/accuracy/accuracy-benchmarking.mddocs/accuracy/accuracy_stubs.mdsrc/aiperf/accuracy/benchmarks/aime25.pysrc/aiperf/plugin/plugins.yamltests/unit/accuracy/test_accuracy_config.pytests/unit/accuracy/test_aime25_benchmark.py
💤 Files with no reviewable changes (1)
- tests/unit/accuracy/test_accuracy_config.py
dynamo-ops
approved these changes
May 29, 2026
FrankD412
approved these changes
May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
aime25benchmark inplugins.yaml(default_grader: math,default_n_shots: 0); scaffold loader raisesNotImplementedErroruntil the full lighteval-backed implementation lands.LightevalExprGrader,expr_gold_metric) introduced in AIP-874.Reference:
trt-llm-benchmark-recipe/src/accuracy/acc_bench_lighteval.py(aime25task)Summary by CodeRabbit
New Features
Documentation
Tests
Chores